Wolfgang Wahlster

In face-to-face conversation humans frequently use deictic gestures parallel to verbal descriptions for referent identification. Such a multimodal form of communication is of great importance for intelligent interfaces, because it simplifies and speeds up reference to objects in a visual context. Natural pointing behavior is very flexible, but possibly ambiguous or vague, so that without a careful analysis of the discourse context of a gesture there would be a high risk of reference failure. The subject of this paper is how the user and discourse models of an intelligent interface influence the comprehension and, production of natural language with coordinated pointing, and conversely how multimodal communication influences the user model and the discourse model. After a brief description the deixis analyzer of our XTRA system, which handles a variety of tactile gestures, including different granularities, inexact pointing gestures, and pars-pro-toto deixis, we present some empirical results of an experiment that investigates the similarities and differences between natural pointing in face-to-face communication and simulated pointing using our system. This paper focuses on consequences of this investigation for our present work on an extended version of the deixis analyzer and a gesture generator currently under development. We show how gestures can be used to shift focus and how focus can be used to disambiguate gestures. Finally, we discuss how the user model affects the decision of the presentation planning component to use a pointing gesture, a verbal description, or both, for referent identification. 3.